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INTERMEDIATE EDUCATICHAL SERVICE AGENCY EVALUATION: 
• LEVELS OF PROGRAM OBJECTIVES AND ASSOCIATED EVALUATION METHODS 

Stan brezek S Alan L. Roecks 

Education Service Center, Region 20 
San Antonio, Texas 

ABSTRACT* . 

A brief sunimary of the evolution of an objectives-based evaluation 
/approach within the context of an intermediate educational service agency 
is presented. A model for conceptualizing program objectives into hier- 
archical levels and focusing evaluation on priority objectives is 
described. Evaluation methods feasible for objectives at each level of 
the model are listed and divided into suggested and other approaches. 
Example priority objectives, evaluation questions, and resulting evaluation 
products taken from ongoing evaluations illustrate the use of the levels of 
program objectives model and its associated methods. 

Paper presented at the Evaluation Network Sixth Annual Conference; 
Memphis, Tennessee; September 29 - October 1, 1980. 
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INTERMEDIATE EDUCATIONAL SERVICE AGENCY EVALUATION: 
LEVELS OF PROGRAM OBJECTIVES AND ASSOCIATED EVALUATION METHODS 

The purpose of this paper is to explain how the "Levels of Program 
Objectives Model" can be used to carry out objectives-based evaluation. 
The context is an intermediate educational service agency. The intended 
audience is practicing program evaluatbrs. 

INTRODUCTION ^ 

Education Service Center^ Region 20 is an intermediate education 
agency serving 50 school districts in a 14-cbunty region of South Central 
Texas; Region 20 is located in the Sari Antbriib metropolitan area. Just 
over three years ago Region 20 established an evaluatibri office, funding 
it as a direct cost written into most grants arid applicatibris and using a 
time accounting system to document evaluation work performed fbr each 
separate funding source;. The office annually evaluates bver 50 programs 

offered to school districts. 

j' 

One of the first tasks in establishing thie evaluatibn office was to 

select an evaluation approach feasible in an iritermediate agency environ- 

■ L_:._._\. _____ _ _ _ _ __ 

ment. in Texas these agencies are highly dependent upbn external furiding. 

_ . _ I ■■ 

Most of the external funding sources requi-re program objectives and an 

associated evaluation plan; Therefore, an objectiveis-based model was 

chosen (Stuff lebe^ & Webster, 1980). Much of the first year was spent 

working with program staff clarifying program objectives. In fact, a 

rather elaborate scheme to focus and document the review of objectives 

was developed. This approach resulted in the Program Objectives Checklist 

(see next page). The concepts of clarity, significance, and measurability 

were useful in reviewing objectives; However, to our disappointment the 

elaborate instrument proved unworkable; 
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PURPOSE: To.docuMMt Cv«lu«t1eo.S*niflcii^:iu99iit1dei do c1ir1ty« tignlflcinct . iod miiiuribdlcy. . Thi 
^m^c ttatrrwnt.ll thi siPonilbilUjf of E^aluitlM Sftrvlcti. ._Ite_reUtloqihiP_Qf ob;jact1v*t 
to ESC*20 9o«1i AAd rtQfonal Msdi li th« roiponilblllty of PUnn1n9 fc Development. 




Objective No.^ 
1. Clear: 



Level 



En«b1ei different readers to picture Viae Iht&it; iliple sehterins; hot iii^lquOus 

2. Signifieaht: ' z ^ 



3. Measurable: 

a. Audience: 

b. Behavior: 

c. Conditions: 

d. Degree: 



• Of genuine ««rth; tangible ret-ults; appears to relate to Center goaU 



tike Others (especially Pattdn (1978)X we found the objectives develop- 
ment process frustrating, time consuming, and definitely not cost effective. 
We constantly asked ourselves "Is it really worth it?** In reality^ we had 
little choice. With nearly ail funding agencies demanding objectives, we 
looked at ways to improve the objective development process. 

Program staff wanted on-the-spot critiquing and rewording of objec- 
tives. They did not want detailed written feedback that took precious time 
from other activities. Informal feedback appeared more effective than 

i 

formal, structured reporting. A tool for more immediate feedback was needed. 
This tool had to meet two heeds. First, it had to produce measurable objec- 
tives. Second, it had to give program staff feedback on how well obj^^tives 
worked together to accomplish priority program goals. The tool that resulted 
was based on a model describing how support prograas, such as intermediate 
educational agencies' programs, assist district instructional programs. 
Using this model and the categories of clarity and measurability 



from the Objectives Checklist the evalaator could work interpersonally 
with program staff to develop objectives. Beddre we describe the model let's 
see* how evaluation staff use the model. The model suggests develdping a 
"student outcome" objective wherever feasible, if this is not feasible, 
often the case in an intermediary service agency environment, "an instruc- 
tional process" level objective would be required. The idea is to have 
"priority objectives" as close to the student outcome level as possible, 
dnce the highest level objective is developed other objectives mast logi- 
cally relate to it at lower levels in the model. For example, an 
intermediary service agency adult education program might have as its highest 

level objective an "instructional process" objective specifying certain 
desirable adult educator behaviors, then the "input level" objective might 
specify paid time to tryout those behaviors. The point is the logical 
necessity of the "lower level" objective to accomplishing the higher level 
objective would be threatened if the lower level objective were not 
accomplished. 

LEVELS OF PROGRAM OBJECTIVES MODEL 
This simple approach (Note 1) focuses on students and the outcomes the 
program intends students to evidence. We term this level stu dent outcome . 
We divide the many hierarchical systems in education into two broad 
classes: support programs and instructional programs . Instructional pro- 
grams are programs whose immediate purpose is to engage the student in 
learning. Support programs are all the other educational programs whose 
outcomes facilitate instritction; however^ they do not directly teach 
students. Obviously the support programs themselves occur at many levels. 
For an example of hierarchical order among support programs consider the 
following:, a university teacher training program, a school district 
Bersohhel selection system* a school district inservice education system. 



and, finally, the instructional program. We believe educational planning 



in fortunate in dealing, with Jufit the two broad classeH of ins truct ionaJ 
^^arid support programs at this point. 

We conceive the instructional program to include input , instructional 

proce ss , and student outcome levels. We use these terms in the broadest 

sense of familiar usage. 

, Student outcome is student performance. The focus of an objec- 
tive at this level, is to define the good which our students will 
evidence in their behavior. • 

• Instruction al pix^cess means whatever occurs between the student ' 
and his instructors or instructional media.. It includes all 
variety of instructional technology. The focus of an objective 
at this level is to define the unique^ powerful instructional 
transaction that is to transfer the rich mix of inputs into 
important outcomes for students. 

« -inpot means the money, staff ^ tithe ^ and material resources 

dedicated to the instructional process to come. The focus of 
an objective at this level is in specifying the "key ingredient" 
--the input that is so vital, e.g.^ a minimum amount of student 
?-^?^>_^^^^ absence would preclude accomplishing student 

outcomes; \ 

Support progra m s have as their on e purpose -producing client learning 

and products wiiich can becoTne l^inpa^tS-tb— the-^nstructidnal program . We 

conceive these support programs ^o include resource , activity ^ and client 

learning/product levelst- These terms are synonyms for th^e inputs^ processes 

and outcomes of the instructional program. Again we are using these terms 

in their broadest, most familiar usage. 



Client, learning/product means the knowledge, attitude, skills, or 
tangible capital gained by the client or school system as a res- 
ult of activities in the support program. The only purpose of 
these gains are as inputs to the instructional program. Support 
program objectives at this level focus on the critical learning 
or product. For example, a training program might focus on a key 
Ihstructibnal competency, e.g., level of classroom questioning. 
A resource proyisibn program might focus on providing a certain 
set of materials (product) to be used in the instructional process ^ 

Activity means the training^ assistance, information dissemination, 
facilitation, or manufacturing which the support program undertakes 
to transform its own resources into learnings and products of 
genuine usefulness to the instructional program. 
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INSTRUCTIONAL PRSgRAM 
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SUPPORT 



OUTCOMES 



RESOURCES 



ACTIVITIES 



CLIENT LEARNING 
OR PRODUCTS 



A network of support programs helps to maintain an instructional pro- 
gram of a district. Programs supporting the instructional program include 
but are not limited to the district central office, community organizations 
such as parent/teacher organizations, local colleges and universities, and 
state and intermediate education service agencies (see figure below). 
Evaluation offices represent support programs which either feed into other 
support programs or directly serve the instructional program. 



Figure 3 

Support Program Inputs to Districts 
' Support Programs 



Local University 
Support Program 



Intermediate Education 
Agency Support Program 



(e.g. preservice 'tra 

m 



ining) (e. g. inser.vice training) 



School District 
instructional Program 



(e;g; curricular development) (e. g. classroom aides) 



District Central Office 



Parent Organization 



Support programs can impact the same part of the district instruc- 
tional program. The classroom teacher, for example, is directly affected 
by both university preservice preparation arid inservice activity spon- 
sored by an intermediate service agency. In sum, the relationship between 
the instructional program and its various support programs as well as 
the interrelationship anKDng support programs themselves is complex. 
Again, we believe educational planning is fortunate in dealing with just 
instructional prograSis and support programs as broad classes of programs. 

This paper focuses on one aspect of this complex relationship. The 
"Levels of Program Objectives" model explains the relationship between 
an instructional program of a district and various programs supporting 
the district program. A variety of support programs are provided by 
intermediate educational service agencies, the context for this paper. 
The remainder of the paper documents how each level of the model and its 
corresponding prioi:ity objectives can be evaluated. 



EVALUATE v!>/ METHODS FOR EACH LEVEL OF PROGRAM OBJECTIVE 

Once we estaGiished the model of program objectives, we began to 
arrive at a consensus of what the most appropriate evaluation method was 
for each level of objective. Just as the objective-based model was the 
most appropriate for our setting but not the ideal of evaluation 
(Stufflebeam & Webster, 1980), so too these methods suggested as appro- 
priate are feasible but not ideal. Each level of program objective has 
a corresponding "suggested" evaluation method. This relationship is 
summarized in the accompanying charts. Objectives corresponding to 
program goals — or priority bbjectives — are evaluated using the associ- 
ated evaluation method. Eithef the "suggested" method — that judged to 
be most appropriate for our setting — or the "other" method if we have 
less resources to devote to the objective. (See chart on next page). 



, LEI OF Pii OBJECTIVE 

INSTRUCTIONAL PROGRAM 

Level l! Stiideht Outcomes 

Priority dbjectlves at this level specify 
:he khowledgej attitude, or skills 
Itudents will. evidence or the products 
:hey will construct, A student outcome 
ibjective is highly desirable for all 
irograms, 



Mil HOD 



SU93ES1 



Student performance measures (in- 
cluding achievement, psychomotor 
and attitudinal), 

Expert revievs of student products, 



Educator judgment of student performancf 
collected in face-to-face or telephone 
interviews, but not surveys. 



Level II: Instructional Processes 

riority objectives at this level specify 
ritical behaviors which should occur when 
student receives direct or mediated 
nstruction. 



Level Illa: inputs from the 

• instructional Program 

riority objectives at this level specify 
ipnts that are "key ingredients" for 
:compiishing- student outcomes. Inputs ; 
'om the instructional program include ' 
iney, time, and materials. 

JO Level iiib: Inputs from the ' 
Support Program i 

iority objectives at this level specify 
pats that are "key ingredients" for ' 
coraplishing student diitcbmes. Inputs' 
om the jupport program include skills 

g^^ ERLC " Inseryice training and I 
assiwraasultatlv e assistaiire. ^ 



iPrc/iding sufficient resources- are 
available, measures of the class- 
room, environment (e.g., observa- 



•Interviews measuring teacher imple- 
mentation including the use of 
instructional materials (e.g., LoU 
interviews - Note 2), 



Educator, reports such as "Teaclier Iraple- 
mentation Checklists" preferably in a 
telephone or face-to-face context. 



Measures of the amount /adequacy of 
inputs, especially the amount of 
time: allocated to instructioii. 



Measures of the amount /adequacy of 
inputs, especially the amount in 
classroom consultative assistance ■ 
received. Measoreji include inter- 
views assessing the adequacy of 
the consultative assistance. 



Educator reports preferably in a telephot 
or face-to-face context, 



Educator reports. such as a Consultative 
Assistance Rating Form carried out in a 
telephone or face-tb-face context. 
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lIvEL OF PM OBJECIIVE 
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_ , , : 




one 


SUPPORT PROGRAM, 

^ Level IV: Client Learnings 

Priority objectives of tliis level specify 
the knowledge, attitudes, or skills 

educators will eVldenpp or thp nrn<1iirt? 
they will construct, ' 


Educator performance measures 
Je.g.j.pte/pbsttests) 
Expert review of educator products 


Educator judgment; on workshops results 
(e.g., noraed Workshop Evaluation Form) 
Follov-up telephone inteniews 


Level V: Activities 

Priority objectives at this level specify - 
the quantity, and quality of support 
prograiii activities siich as training,, 
consultative assistance (not on-site), 
infdnnation dissemination i and coordlna- 
tlbn/llaison. 


Hours of service measures (e.g., 
educator hours). 

Educator judgments on quality of 
services preferably in a tele- 
phone or face-to-face contexts 


Participant observation 

Educator judgment on workshop conduct 

•M- ■■ —M ■ --■ i"""f " ' 4 ' • — V 

(e.g., nonned Workshop Evaluation Fen) 

* 


Level VI 1 , Resources 

Priority objectives at this level specify 
tangible products (or. their revision) 
3uch as instractional_iiiodnies, _fii5s, _. 
dts, or evaluation findings reported in 
leeds ass^ssients, interim memoranda and 
final reports. 

i.2 ■ 


•User tryout (piloting parts of the 
product with flsersj 

•Using-EvaiuationJata-iorm: - a . 
format for reporting major evalua- 
tion findings, discussing- these 

:with project staff, and arriving 
at appropriate action statements, 


•intrinsic evaluation (Scriven; 1967) 
methods such as documenting content of 
product, reviewing product objectives, 
logical analyses of the relationship 
between product activitifes and client 
learning objectives, and expert reviews, 

1 

1, 

13 " 

y ■ • 
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tet's first look at the instructional prbgraih. The suggested method 
for student outcome objectives is a measure of student performance. We 
. do not often have ah opportunity to use student performance measures 
because our agency's programs are primarily directed toward teachers. 
Therefore, the instructional process objective is a common one and we 
suggest evaluating it by interviews measuring teacher implementatiori of ' 
critical features of a program. The specific technique we rely on is an 
adaptation of the Levels of Use focused interview (Note 2) developed by 
Gene Hall and associates at the University of Texas at Austin. For 
input level objectives, we suggest measures of amount or adequacy of the 
input. Time measures can be particularly. revealing. 

Looking at the support program, we suggest educator perfotmance data 
is most appropriate for the higiest level, client learning. This would 
include pbsttestihg of trainees. oh objectives. We are not always able to 
convince project staff of the necessity or even desirability of post- 
testing. Pbsttestihg may alsb be impractical in ifl-dfefined projects^ 
because we do not have the, time to clarify training objectives and develop 
items. In such cases a ribrmed workshop evaluation form is used. At the 
activity level, we suggest hours of service prbvided tb educators in such 
activities as training, consultative assistance^ infbrmation disseminatibii^ 
and coordination/liaisbri. This is often more revealing than the educators' 
judgments on the quality of services. A particularly useful "bther** 
method at the activity level is participant observation. Evaluation staff 
arev encouraged to attend some aspect of project activity. In some cases 
evaluation staff actually go through project training. Finally, at the 
rfisoufce level, bur most powerful method is to document use of information 
with the Using lEvaluation Data form. The form provides a format for 



reporting major evaluation findings, forces an occaisibn for formally 
discussing th *se with project staff, and allows for recording action 
•taken. 

The application of "Levels of Program Objectives" to a program 
evaluation setting is further illustrated in the appendix. Example 
evaluation questions and products corresponding to each level of bbjec-. 
tive are presented for those evaiuators wishing to learn more about the 
model. 

PRIORITY OBJEeTIVES 

Educators rarely have time to plan t>rograms adequately before conduc- 
ting them; Educator?? face so many immediate problems they perceive they 
must act immediately; Unfortunately, they do not reflect on the cogency 
of their actions. When properly used, the "Levels of Program Objectives" 
model causes educators to esiamine the consequences of their action. The 
structure of the model encourages objectives to be written at the highest 
possible level. Priority objectives then are identified. 

Priority objectives correspond to the goals of the program. These 
are the objectives which must be achieved. If they are are not, the board, 
advocacy group, special interest representation, or funding agency will 
surely question program funding! They are so vital that program staff too 
believe they must be accomplished. Iif our setting a $50,000 program may 
have three, objectives, but just one priority objective. 

The number: and scope of objectives determines the allocation of evalu- 
ation resources. Practical consideration dictates that every objective can nc 
be ideally evaluated. Seen another way, not ^ ail writ ten objectives are 
priority. Evaluation resources are first dedicated to priority objectives; 
remaining resources are spread .among other program objectives. Often^ the 
level of evaluation method 'selected corresponds to a lower level bb^iective; 



(A client learning objective (IV), for example, is evaluated usihjg; methods 
suggested for the lower level (activities objective (V)). The model then 
can be applied to the real world of projgram evaluation where there are rarely 
enough resources to get the job done. 

BEYOND THE EVALUATION. OF PRIORITY OBJECTIVES 
The reader will have noticed the heavy summative flavor to the "Levels c 
Program Objectives" and evaluation methods suggested. Howevei^i we have come t 
believe summative information on accomplishment of objectives must be comple- 
mented by formative information which responds to specific concerns of program 
staff. Furthermore^ extensive involvement of evaluation staff in the program 
is essential if action is to result from evaluation findings. 

Our Evaluation Services unit meets ±tk responsibility for Summative 
evaluation by selecting priority objectives for evaluation. Evaluation staff 
usually collect data on the amount of service provided under priority objec- 
tives, and, for some priority objectives, data on the quality of that service. 

After making sure that the priority summative question(s) will be ans- 
wered, evaluation staff and program staff focus on concerns of program staff; 
Normative evaluation includes evaluation questions that are of present concern 
to the staff and can therefore be included in an evaluation plan, as well as 
allowing resources for additional evaluation work which emerges in the 
cdurse of the program year; As time has gone on we have been shifting more 
and more to formative and responsive evaluation. 

CLOSING COMMENT 

In closing, several things need to be said about the model. First, it 
seems to work in our setting. We have been able to successfully classify 
over 800 objectives using the model. Presentation of the model at this confe- 
rence is bur first attempt at dissemination. Second, the concepts of the 



useful in breaking down the oh-sd-connnon coirimunication barriers^ in objec- 
tives clarification. Third^ the model has provided a unifying force in 
bur newly developing evaluation office. Explaining our evaluation approach 
has been difficult. The model provides an excellent starting point; 
Finally^ we believe the model has the potential to be an effective training 
guide in ari objectives-based setting. Our goal this year is to train staff • 
with it. If you're interested^ we can report the results of our "training 
activities to you next year. The conference will be held in our hometown of 
San Antonio. 
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INSTRUCTIONAL PROGRAM: Level I^ Student Outcomes 



Priority objeotives at this level speoify the knowledge-^ attitude-^ or skills 
s^wrfents wtZZ evidence or . the products they will construct. A student outcome 
objective is highly desirable for all programs. 



- INSTRUCTION A L PROGRAM 



iMPfn-c 4lr*5TRUGTI0NAt^ ^STUDENT 

iPiruia w PROCESSes^ I^OirTCOMES 



SUPPORT PROGRAM^ 



RESOURCES ^ACTIVITIES- — |CLIE^^r LfAHNiNG 

OR PRODOCTS 



Program context: 



Evaluation method: 



Priority objectivei 



Evaluation question: 



A .model program for deaf-blind infants funded under 
I\r-e of_Public Law 93-380. . 

the program component illustrated by the objective below 
was a direct instructional program. Inputs bf 1.5 hours < 
per week. were specified; no instructional process was 
defined. 

Student performance measure: 

The Learning Accomplishment Profile documents developmental 
rate by observation. 

Given an averag&< of 1.5 hours per week of infant-parent 
training, at least two-thlrds_ of six deaf-blind infants, 
9^2, will have a greater deyelbpraental rate in specified 
^^'^^^i^^ instruction than their rate one mpnth after 
entering_ the program, as evidenced by a 10% increase in 
developmental rate on the Learning Accomplishment Profile ' 
for infants (LAP) . 

How many deaf-blind infants have increased their develop- 
mental rate by 10% in specified areas of instruction? . 



Evaluation product 



See example 1 on next page 



EXAMPLE i: 



kEVEb I : Student Outcomes 

Suggested method: Student PERFORMANeE measures 



Evaluation Question 2: How many deaf-blind infants have increased 

their developmental rates by 20% in specified 
areas of instruction?'^ 

Pre- and post-service assessments using the Learning Accomplishnient Profile 
(LAP) have been administered to the six students receiving regularly sched- 
uled training (1-^2 to l*f Srs/wk). The table below presents both pre- and 
post-tAP data for these six cases: 



Change in Developmental Rates for DB Infants ». 







Pre- 


-Service 




Post- 


-Service 




Durihg-Service 




DOB 


( in^Sos . "y 


Levei 


Rate^ 


(in*S8s.) 


Level - 


Rate^ 


Rate^ 


>S37 


4/12/79 


6;0 


i.4 


0.23 


10.5 


3;5 


0;33 


0.46 


528 


• 8/3/77 


21.0 


2.0 


0;10 


32.4 


2.1 


0.06 


OiOl 


503 


3/27/77 


19.1 


5.9 


0.22 


34.8 


6:;i 


0.18 


0.10 


535 


; 11/12/76 


32.3 


i.5 


0;05 


41.0 


i;7 


o:o4 


0.02 


254 


4/14/77 


25.0 


1.6 


0.06 


37.0 


• 1.8 


0.05 


0.02 


539 


8/21/77 


24.9 


6.0 


0.24 


29.8 


6.7 . 


0.22 


0.14 ? 



a The pre-and post-service developmental rate reflects months of development 
per month of chronological: age; 



t The during-service deveiopmentai rate reflects change in developmental 
level pre to post divided by months of service. 

One student has demonstrated an increase in overall deveiopmentai rate as 
well as in all areas of instruction (see p.i7 ); this student's overall 
developmental rate has doubled during trainitig. One of the five remaining 
students showed a definite increase in developmental level, and the rest 
were able to maintain -their developtnental level. 

Because there is no comparison sample for this population, it is difficult 
to identify an appropriate criterion for these infants. This year, the ? ame 
goals were tentatively set for the deaf-blind^inf ants as were set for the 
visually handicapped infants. The criteria seemed appropriate for the VH 
infants but not for the DB infants (see pp. 18-20 for a report of VH student 
progress) . 



*The infant-parent trainer sees herself as providing services affecting ail 
six areas of developmenti 
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INSTRUCTIONAL PROGRAM: Level II, Instructional Processes 



Priority objectives at this level specify critical behayioy^s which rAiould 
occur when a student receives direct or mediated instruction. 



nm PROGRAM 



INPDTS- 



I NStRUCT 1 0NAIJ_^ S TUDEfiT 
ES I ▼'OUTCOMES 



PROCESSES 



RESOURCES 



ACTIVITIES f CLIENTT LEARNING 

OR PRODUCTS 



Program context: 



Evaluation method: 
Priority objective: 



Evaluation question: 
Evaluation product: 



An Elementary and Secondary Education Act, Title IV-C, 
model program whose purpose was to train teachers in 
innovative mathematics methods. There was no- student 
outcome objective. . Support program objectives in- 
cluded developing a module (resource level) and training 
(client learning) . An instructional program input 
?-?Y??:_°^4^^_^^^^ specifying the provision of consultative 
assistance was present. 

A Level of Use interview measuring teacher implementation 

Between December 1, 1979, and May 30, 1980, participants 
will develop and implement a plan to Improve mathematics 
instruction in at least two areas where training was 
received. The plan will include objectives, activities, 
and evaluation. Accomplishment of this objective will 
be evidenced by: (1) plans on file at ESe-2b for at 
least 70%_ of the participating teachers, which will 
include objectives^ activities, and evaluation, and 
(2) responses to questions asked during Levels of Use 
interviews with a sample of teachers indicating that 
50% are 'implementing their plan. 

Are 50% of the teachers implementing their plan? 

See example 2 on next two pageis. 



EXAMPLE 2: Level II: INSTRUCTIONAL PROCESS 

SUGGESTED METHOD: INTERVIEW MEASURING iMPL^ENTATION 



TITLF.: 



DATE: 

DOCUMEr^T^riL'MHER: 



EV:^!:l;ATiOK S£RVK:^:f ^^^M5RANDUM 

Math Attitude Levels of Use Report 
John Andrews 
June 12, 1980 
79:203 
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PROGRAM OBJT^CTIVES ADDRESSED 

Objective 3.0 of the 1979-80 Teacher. Attitudes -^nd Competencies in Mathematics 
proposal states that 50% of participating teachers will be implementing new 
strategies to improve their mathematics instruction in at least two of the four 
areas in which training was provided. 

Objective 4.0 states that 75% of_the participating teachers who received technical 
assistance will consider it to be helpful. 

Both of these objectives were assessed by conducting interviews with a sample of 
teachers trained. 

FINDINGS ON ACeOI^Li SH ME NT OF THE OBJECTIVES 

1. A total-of 76% of the teachers were found to have implemented new strategies 
in at _ least two areas. Ail those who were iraplementers were able to use the 
activities smoothiy^in the classroom. This is probably due to their selecting 
and using activities that were most _ consistent with their own teaching style. 
The non-i raplementers did not use the new activities becatise they were satis- 
fied with what they were already doing. 

2. By the time of the interview, 9d%.of the teachers had received a technical 
assistance on-site visit. Of these, 63% believed the visits to be useful. 
Teachers were most likely to believe the visits were useful if the consultant^ 
demonstrated, with the students, some activity or procedure. 

ADDITIONAL FINDINGS • . 



The tnacliars were most likely (81%) to have used new strategies in the area 
of Concrete Activities. . However, few teachers seemed to have internalized the 
Concretc-Pi ctoral-Abstract continuum. Rather, they saw the Concrete Activities 
as isoilated tools they could use. 



2. 



The teachers were least likely (33%) to use the new strategies in the area of 
Problem Solving. This was, apparently due to the fewer number of specific 
Problem Solving Activities and materials suggested to thetn. Also, there may _ 
not have been c^nough workshop time to adequately cover the subject. Tc^achers 
had difficulty translating, on their own, theory and concepts into classroom 



Reai iife iTiat'^i acLivicies were the one? most frtcui^ntly used. 



4i tess than haif of the teachers wc-re lisin^: uhe l.-^nei-.c^'e of M:-th Ac tiv: r i i^t: 

5. About 15% of the teachers had already completed the teacher developed activites 
they were suppose to develop for the project. Abbut_70% indi cated' that thoy 
planned to do so. The inost frequently given reason for not Having completed 
the activities was lack of tirae. 

6. The strong points of the inservice training were seen as being the multitude of 
ideas that' were presented and the opportunity to share with other teachers. 

7. The weak points of the training were seen as (1) too much emphasis on theory, 
^2) scheduling the_workshops after school^ (3) scheduling the workshops in the 
ndddie of the school year , (4) the lack of time to make materials, and (5) 

not having the workshops separated for Primary and Intermediate teachers. 

8. The teachers' own attitudes toward math apparently improved. 

9. A number of teachers seemed isolated from feedback on the quality of their in- . 
struction. An important role for the project staff was found to be -reinforcing 
and encouraging teachers for the good work ; they were already doing. 



INSTRUCTIONAL PROGRAM: 



Level III, Inputs' 



P'Hority objedtives at thio Xevet specify znfnAt^ are "^^y ingredient-3" 

for accomplishing student QutcQmes. . Inputs inalude money ^ time^ materials from 
the instructional _ program itself and skills acquired^ products, or consultative 
assistance from the support program. 



I^^STRUCT-t 



' PROCESSES W o/TCCrncS 



SmilJ PROGRAM^ 



... i 

FSSOURCES lACTIVlTjES— ^CLIENT LEARNING 

OR PRODOCTS 



Program context: 



Evaiuation^method : 
Priority objective 



Evaluation question; 
Evaluation Product: 



Another P. L. 93-380, VI-C mod el program for deaf- 
blind, students this time adolescents. Thiamins true- 
tibnal program input level objective was accompanied 
by a priority objective at the student outcom^ level, 
a measure of work skills. No support program/ priority 
objectives were written. The instructional p'rocess 
wais not defined in an obj-ictive. 

Amount of titne allocated to the instructional process. 

Between-July 1, 1979 and June 30, 1980^ four deaf-blind 
adoiescents-will receive work evaluations — three days 

four others, work adjustment training — 160 hours 
each,_and two others, paid work placement, 300 hours 
each as evidenced by (1) documentation of amounts of 
experience and (2) case-by-case interviews with the 
primary work trainer (aide) and supporting daily Work 
Training Center Performance Records; 

^^^B,^ 640 hours of work training been provided? 

See example 3 on next two pages 



EXAMPLE 3: LEVEL ili: INPUTS 

SUGGESTED METHOD: AMOUJilT' OF TIME ALLOCATED 

Evaluation Ouestibh 2 ffas 640 hours of Dork training been provided? ^ 

As of March 30, 1980^ a total of 505.2 hours C^02.2 work hours, arid 103 
live avay experience hours) of training had beeri provided to the 14 
clients "served by the proj ect (see Table 1). Based on the average amount 
of service per quarter and the scheduled live away for June 1-6^ it can 
be estimated that the project staff will probably come close in providing 
the 640 hours of training. 

the evaluation question states work training, however^ the prcjecx: staff 
view the teaching of _ independent living skills and work training skills as 
same^ -This view is substantiated by a high cbrrelatiba r ^ .86 
of-the S an - Francisco Vocational "Competency Scale used to measure students 
P^^ J^^ PP^^zS^o^S^ ^i'^pr^ skills with the Camelot Behavioral Checklist 
used to measure students pre to post growth in indeperident living skills 
(see Evaluation Qbestiori 5). 

The mean amdimt of service for the 13 clients.with service was about 39.0. Ther 
is high Variability in the. amount of service. per client . One client received 13 
hours (/?6504)i and three clients received what might be interpreted as a low 
amount of //6527, j?6506^ J?6524 , and one client (//65i8) received no service, but • 
this was due to this client being out the first five months of the prbgrani for 
major eye surgery. 

The December^ worfc trairiing hours were low because of a problem that arose with 
transferring the Goodwill ERS students to community work sites. Since solving 
this provlem the primary work trainer has worked with clients of the project on' 
small group rotating basis. , 

Site visits to Goodwill North substantiated the efforts of: project staff in 
providing work adjustment training. A formal inte^T^iew-wa^condu^^^rf on 12/12/7 
with the Head of Housekeeping at the Downtown Holiday inn. T^w^lnterview veri- 
fied the appropriateness of this setting for work adjustieSt training 
(see pg. 13 ) r 



EQ 2 



Table 1 
^rk^raiiiing Hours* 



^3/ 




..ppl»»efttil tfaining i„ the JSfskml'"' '""^ 



SUPPORT PROGRAM: Level iV, eiient Learninj5s 



eduQuiors will evidence-^ or the products: they will Qonsiruot-. 



INSTRUCTIONAL PROGRAM 



INPtJTS- 



i INSTRlXTJONAU^i STUDENT. 
" PROCESSES ~ OOTCCrtS 



SUPPORT Pf^RAM^ ^ 





1 


RESOURCES IaCTIVITIES- 


-fCLlENT LEAW^lll^ 
Oft PHODUCTS 





Program context: 



Evaluation method: 
Priority objective: 



Evaluation question: 
Evaluation product: 



An early childhood education for the. handicapped in- 
centive grant, funded under Public Law 94^142. - The 
program contained an instructional program objective 
at the- instructional process level addressing imple- 
mentation of mastered learning. The support program 
objective is given below. 

Educator performance measure (Posttest) 



Between January 30, 1980 and August 31, 1980^ fifty , 
additional personnel involved in early childhood, 
programs will have the necessary skills and knowledge 
to conduct effective parent invblvemeht groups, 
Accbmplishmeht of this objective will be evidenced by 
80% of the participants deihbhstratihg mastery on 
(a) the Pbsttest (80% correct) and (b) the Group Skills 
Checklist (60% correct^ including summarizing). 

Did 80% of the participants demonstrate mastery? 

See example 4 on the next two pages. 



EXAMPLE 4: LEVEL 4: CLIENT LEARNINGS 
SUGGESTED METHOD: ; POSTTEST 



ECE-H 

OBJ J 3 (a) 



PRE AND POST-TESTING- 

Background 

Objective 3(a) of the_ proposal specifies that fifty early childhood 
personnel will have the necessary skills and knowledge to conduct parent 
in-trblvetnent groups as evidenced by 80% of the participants scoring 
80% or higher on the Posttest. 

Pre and Posttest items used were the same as for the previous year (see 
Attachment) . Two additional posttest only items were added this year 
as a check to verify if participants are learning more than the 
"required" materials. 



A random sample of ten- cases wete- chosen by the evaluator to see if 
the test scoring procedure (as scored by the project manager) was 
reliable; . Jm interrater reliability of .98 was obtained; 

Nearly all (48 of the 55 participants, 87%) had completed the pre and 
posttest. . • ' 

Findings / 

Objective. 3(a) was reached in thai: 94% (45 of 48) of the participants 
scored 80% or higher oh the Posttest. 

Nearly all (94%) of the participants scored 80% or higher on the two 
additional four part items indicating the participants were beyond . 
studying only for the Pbst;t^st. 

On the average, scores rose from 17 on the Pretest to 94 on thie 
• Posttest. 

Table 1 1 D istribut io n of Posttzegt Scores 



Posttest Only items 

1979 1980 (maximtim 8 pointi) 



Score 


J_ 


_% 


JL 


_% 


Score 


,_1 


_i 


70-79 : 


2 


5% 


3 


' 6% 


4 - 6: ■ 


3 


6% 


80-89 : 


3 


n 


5 


10% 


7: 


11 


23% 


70-99: 


16 


39% 


24 


50% 




34 


71% 


100: 


20 


49% 


16 


33% 









GROUP SKILLS 



Background 

Part b of ebjecttve 3 in the proposal states that at least 80% of the 
participants^ili demonstrate use of at least 60% of the groUp skills 
including summarization; 

A sample of ^2 participants were randomly selected to conduct simulated 
parent meetings of approximately 20 minutes each. A project . staff member 
observed each participant for demonstrating any of ten group skills (see Table 3). 



Findings (See Tables 1, 2, and 3) 

. About 45% of the participants demonstrated 60% of more of the group 
skills including summarization. 

The number of group skills demonstrated in this year's longer 
sessions (about 20 minutes instead of 10 minutes) greatly increased 
over last year's with the exception of "task setting and obtaining 
committments." 




HlIIM'Oiri* nUKJUAM: IxiVol V, Ac:lIvlLloH 



progt'am 'a'ativltieii .cuoh as training ^_ _'conQul in t^^^^ ass^intanna-y (noL on- 
information disseminatioh, and aoordinution/liaison. 
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INSTRUCTIONAL ?mm 



I NPt/TS- 

I 



1 1^6TRUCTI0NAU:::^^ STUDErfr 

^ PROCESSES T OUTCOMES 



1 



SUPPQU PWXfW^^ 



RESOURCES ^ I ACT 1 VI T 1 £b — }| CL I ENT LEARNING 

OR PRODUCTS . 



Program context: 



Evaluation method: 
Priority objective: 



Evaluation question: 



A state and federally funded adult education 
cooperative program. It involves multiple agencies 
over. a wide geographic area. No support program client 
learning objectives were written. Instructional 
program input level objectives were present^ but 
higher level objectives were not. 

c- _ ^_ 

Educator judgment on the quality of services 

By May 30, 1980^ all Adult_ Education staff will have 
completed the required twelve hours of in-service training 
provided. Training will consist of the f ollx5wing : 

(a) A series of courses designed to provide teachers 
with an opportunity to select training appropriate 
to their needs and-^desires^. ^ _ 

(b) Additional-workshops that might be presented by 
universities, state or federal projects, and teacher 
organizations as applicable to the adult learner. 

(c) Undergraduate and graduate program offerings by 
area colleges ^and universities for college credit. 

(d) Individualized training (films, modules,' independent 
study) applicable to the adult learner, _ 

Accomplishment of this objective vrLll be evidenced by at 
least 50% of the Adult Education instructors responding 
favorably to selected items on a workshop evaluation form 
or during an individual interview^ depending on type of 
inservice completed. 

Did 50% of the instructors respond favorably (pption"4" 
or "5") on items 7 and 8 of the .Workshop Evaluation Form. 



Evaluation product: 



See example 5 on the next two pages. 



^ WEP, OBJ. 3.2 



EXAMPLE 5i Level V: ACTIVITIES 

SUGGESTED METHeD: JUDGMENT ON SERVICE QUALITY 

WORKSHOP EVALUATION REPORT 



RKSHOP TITLE: Teaching Readking Skills DATE OF WORKSHOP: 4-23-80 

NSULTANTy ROJECT: Adult Education 

ir 

i\LUATION STAFF PREPARING REPORT Ethel Turner REPORT DATE: 5-6-80 



WORKSHOP DESCRIPTION 

MAJOR OBJECTIVES (SUMMARIZED) : ' 

f9z^91^^ three primary areas for teaching reading to adults: (1) methods for 
determining reading levels, (2) suggested graded materials to be used in a reading 
program, and (3) methods for evaluating progress. 

LENGTH OF WORKSHOP: 3 hoars 

OUTSIDE CONSULTANT: none 

AUDIENCE: Number (%) completing WEF : 16 (100% 

Number attending: 16 ' 

Type of Participants: GED^ APE, and ESL teachers 

Teacher aide - 1 

_ :/ 

miNGS ON THE IMPACT OF THE WORKSHOP 

analysis was made of the responses to the "butcbme" items which describe the impact 
the workshop. These results are presented b^^w. 

A; Ail - (100%) of the respondents indicated moderate of isubstantial new 

learning; Forthis question this workshop ranked 1st of the 29 Adult 
Education Workshops evaluated to da^te, 

B. Most (81%) of the respondents indicated they wiii apiply their learnin|^^ 

which colild result in moderate or large increases in effectiveness. For this 
question this workshop ranked 7th of the 29 Adult Education Workshops 
evaluated to date. 
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........ .. i _ . . ...... 

Participant comments indicating the beneSiciai aspects of the wdrkshbi) iricludihg- 

(1) : having a consultant presenting the-material who was enthusiastic, well 

prepared, and very knowledgeable- about_the subject (11 comments), 

(2) reviewing the Adult Education reading materials_ (8 comments) , 

I (3) evaluating and administrating reading tests; determining reading level 
(3 cpmmehts) , 

(4) receiving an overview of the Goodwill Reading Program (2 comments), and 
/ (5).> teaching basic reading (1 comment). 

Two participants felt the workshop would be more useful if they had received a list 
of addresses/publishers of the books and materials used by the consultant. 

Participants provided suggestions for future workshops. They included: 

(1) working with adults - counseling arid guiding (2 comments), 

(2) curriculum organiEatidn arid management - e,g, developing a program, 
deciding it should be individAjalized, teacher-directed^ etc. (2 comments), 

(3) teaching math to adults, arid 

(4) teaching reading to adults. 

Two participants commented on their needs as an A^ult Educator. Orie commented 
''1 had a problem this school-year in A. B.^ not kridwing what adult materials 

and their sources were avaiiable^_ piis "workshop helped, but tndre iriformation 
would be helpful." _The other_respondent felt they needed to have their basic 
curriculum and materials "uniform ail oyer the city." 

SUMMARY AND SUGGESTIONS 

In, view of the high percentage of "learning" and above average percentage of 

application" to this workshop, and the favorable comments from the_ participants ; 
it may prove beneficial if this workshop was repeated next year for new teachers. 
One participant's comment might be a reflection on why this workshop was well 
received. "Use people who are presently teaching, in workshops. We. need practical 
ideas not just theory." 

EtT/es 




APPROVE D, : 

Stan Drezek^ Senior Evaluation Manager 
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SUPPORT PROGRAM: tevei VI," Resoarces 



PrioY*iiy obgeciives at this level BpeaiSy iangibte product:^ Cor their revision) 
such as . instructional moduteSy fitrriyS brochures ^ or information packets and 
evaluation findings such as needs assessments-^ interim mernoranday or final 
reports. 



iNSTRUCTim PRQGfW- 



iNprrrc 4lf*STRyGTI0NAl^4STUD£Wr 

' ^ ^ PROCESSES ^ OUTCOMP! 



RESOufiCES- 



lACTivlTIES— I- CLJENT. LEARNING 
OR PROnXTS 



Program context ; 



Evaluation method: 
Priority objective: 



Evaluation question: 



A Department of Education Teacher Center program. 
The majority of objectives were at the support program 
activities level. This priority objective at the resource 
level was a needs assessment. Findings influenced the pla 
ning of support program activities. 

Using Evaluatibh Data form 

By August 1980, the evaluation staff will have surveyed 
a 5% random sample of teachers and administrators in 
the area served by District_XX, Texas State Teachers 
Association regarding (i) the perceived. instructional _ 
needs of students in specific skill areas that are basic 
to aii_ academic disciplines and (2) specific. inservice 
needs of the teachers. Accomplishment of this objective 
will be evidenced (1) by a final report in project 
files and (2) at least three recommendations for program- 
matic improvement made jointly by project and evaluation 
staff as recorded on the Using Evaluation Data Form . 

Have the needs assessment findings been used to 
recommend specific workshops for presentation? 



Evaluation product: 



See example 6 on the next two pages. 



FlilNS ^: 



.__ KEEPS ASSESSMENT 



Survey responses from a stratified randoin sample of 
W teachers and administrators regarding their 
specific inservice needs show 

1. ''Motivation-yours.and theirs" .was indicated by 
both elementary and secondary teachers as a 

- l"s«vice topic (551 and M respectively). 

2. Additional topics for inservice indicated as a 
"^^S^ "«*ed for elementary teachers (n=241) were 

: 1) Developing subject materials appropriate to 
the needs, abilities, and.interest of students 
[M] and 2). Creating classroom materials (52X). 

3. For secondary teachers (n=216) additional topics 

Handling stressful situations in the 
classroom (50X) and 2) Challenges for the Gifted/ 
Talented students {kU], 



Respondents asked,fbr_ additional inservices iii 
matheniatics specifically math games, developing 
mathematics learning centers, use of computers 
and calculators, metric education, Chisanbop-, 
individualizing math, reieaming basic math 
skills to teach slow learners, and alternative 
approaches to math. 



Action means concrete policy, procedures, decisions, or 
".Vonctlon;' niay be justified but should be explained for 



mm m 



mm 
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The Needs Assessment data was used by project 



^\ 

_i"::Pl2nning 1979-80 spring and suSer %^ 

workshops, ^and is being used to develop theJ980- % ' 
.^.^N^le- A summary of the Naeds.Assessment % . 
results was also reported jo the area teachers/admin- i 
istrators in the Teacher Center Newsletter; "^^ 



1.- 3. 



Specifically, during sprlng.and stinier 1980 in 
answer^to teacher/administrator inservices needs 
the follDWing Insenices (and number of) were 
presented" 



Motivation (6) 
Stress. (3) 
Creating classroom 
materials (^6) 



- Developing subject mate- 

rial priprbprlato to 
the needs and Interest 
of the student (i) 

- Challenging the secon- 

dary gifted/talented 
student (1) 



^. Hath inservices presented included- 



- CHAMP .d) 

- Chisanbop (3) 

- Progressive trends in 

' Teaching metric systems 

for bilingual teachers 
(1) 

assignments. ' " • 
the record; 



- Sbntessorl method for 

teaching math (1) 
" Math games 

- Elementary math:- an 

old approach made new 

(1) 

\ 35 
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5. Teach|rs also Indicated a need for additional 
training in teaching oral language. and language 
cbiiipbsitibri. Specificallyi Ihservices suggested 
Included speech and language development, improv- 
ing listening skills, develdpihg conversational 
skills, teaching grammar, and vocabulary develop- 



6. In order to better teach reading to their students 
teachers iieed additional training in teachiti|^ 
reading iii the content areas, teaching reading 
to the iiilddle school student, reading coinpre- 
hension, activities for reading groups, and 
managing a reading program. 



5.- 6. Inservices presented on oral lariguage/larigiiage 
composition, and reading included— 



- Reading iri the content 

area (4) 

- Managing a reading pro- 

gram (3) ' 

- Teaching writing (1) 



- The I's of reading 
. (1) 

- Reading the fin;incial 

•T)ages of the news- 
paper (1) 



Action means concrete policy, pfOcedureSj decisidhs, or assignment's 



